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1  Introduction 

RMIT  participated  in  the  2009  Web  Track  tasks.  Our  submissions  utilised  the  Zettair  search  engine1 
to  index  and  search  the  Category  B  subset  of  the  ClueWeb  collection  used  by  the  Web  Track. 

The  Web  Track  was  composed  of  two  tasks,  a  traditional  adhoc  retrieval  task,  and  a  new  diver¬ 
sity  task  where  participants  attempted  to  retrieve  documents  covering  a  range  of  sub  topics  for  each 
query.  Sub  topics  were  not  provided  with  the  queries. 

Our  experiments  utilised  the  well  known  measures  Okapi  BM25  and  language  modeling  with 
Dirichlet  smoothing  for  the  adhoc  task.  For  the  diversity  task  we  attempted  to  improve  the  diversity 
of  query  results  by  minimising  the  number  of  documents  returned  for  a  single  domain. 

2  Description  of  Runs 

Runs  were  generated  using  a  customised  version  of  the  Zettair  search  engine  which  was  adapted  to 
deal  with  the  large  scale  ClueWeb  collection.  All  runs  used  the  default  light  stemming  option  in 
Zettair  that  removes  the  suffixes:  -e  -es  -s  -ed  -ing  -ly  -ingly,  and  replaces  the  suffixes  -ies,  -ied  with 
-y.  No  stopping  was  used. 

2.1  Adhoc  Task 

For  the  adhoc  task  two  runs  where  submitted: 

•  RmitOkapi :  The  top  1,000  retrieved  documents  ranked  by  the  Okapi  BM25  similarity  mea¬ 
sure  (Sparck  Jones  et  al.,  2000).  Parameters  were  left  at  the  default  Zettair  values  with  Kl, 
K3,  and  B  set  to  1.2,  oo,  and  0.75  respectively. 

•  RmitLm:  The  top  1,000  retrieved  documents  ranked  by  a  Dirichlet  smoothed  language  mod¬ 
eling  measure  (Ponte  and  Croft,  1998,  Zhai  and  Lafferty,  2004),  with  the  //  parameter  set  to 
1,500. 

'Zettair  is  available  under  a  BSD  License  from:  http :  //www.  seg.rmit .  edu.au/zettair 
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Table  1:  Stat  mean  average  precision  and  stat  mean  nDCG  for  adhoc  runs. 


Run  Label 

StatMAP 

StatMnDCG 

RmitOkapi 

0.1558 

0.3222 

RmitLm 

0.1686 

0.3113 

Mean  per  topic  median 

0.1539 

0.2956 

Mean  per  topic  best 

0.4304 

0.6091 

Table  2:  Alpha  nDCG  and  intent  aware  precision  scores  for  submitted  and  baseline  diversity  run. 


Run 

alpha-ndcg 
@5  @10  @20 

@5 

IAP 

@10 

@20 

RmitLm 

RmitDiv 

0.103  0.147  0.188 
0.097  0.157  0.193 

0.055 

0.046 

0.075 

0.070 

0.084 

0.072 

2.2  Diversity  Task 

For  the  diversity  task,  one  run  was  submitted  ( RmitDiv )  where  the  top  1 ,000  retrieved  documents 
ranked  by  a  Dirichlet  smoothed  language  model  similarity  measure  were  returned  after  filtering  the 
results  such  that  the  domain  name  of  a  document  appealed  no  more  than  once  in  a  document  list. 
Where  a  document  from  the  same  Internet  domain  appeared  more  than  once,  the  highest  ranked 
result  was  retained  in  the  list,  and  all  other  documents  were  removed. 

The  rationale  behind  this  approach  is  that  documents  returned  from  the  same  domain  have  a 
higher  likelihood  of  discussing  the  same  sub  topics,  as  opposed  to  documents  of  differing  domains 
that  may  cover  different  sub  topics  of  the  query. 

3  Results 

Global  measures  arc  not  yet  available  for  the  Web  Track  limiting  the  analysis  of  our  results.  Our 
system  performance  in  comparison  to  the  median  and  best  results  for  individual  topics  is  discussed 
below. 

3.1  Adhoc  Task  Results 

For  the  adhoc  task,  the  stat  mean  average  precision  (StatMAP)  and  stat  mean  normalised  discounted 
cumulative  gain  (StatMnDCG)  results  for  each  of  the  submitted  runs  arc  presented  in  Table  1.  The 
two  similarity  measures  resulted  in  runs  of  similar  accuracy. 

For  each  topic,  the  best,  median,  and  worst  values  of  stat  AP  and  stat  nDCG  over  all  submitted 
runs  were  available.  By  averaging  these  results,  the  mean  median  and  mean  best  values  were 
calculated  and  presented  in  Table  1.  As  expected,  having  submitted  baseline  system  runs,  the 
overall  accuracy  of  our  submissions  is  close  to  that  of  the  median  per  topic  average  score,  and 
significantly  lower  than  the  best  score  average. 


3.2  Diversity  Task  Results 


A  single  run  was  submitted  for  the  diversity  task.  Table  2  shows  the  mean  scores  for  alpha-nDCG 
and  intent  aware  precision  for  our  RmitDiv  submission,  as  well  as  the  scores  for  a  baseline  run 
RmitLm  on  which  the  submitted  run  was  based. 

The  table  shows  mixed  results  with  a  minor  improvement  in  accuracy  measured  only  with  alpha- 
nDCG®  10  and  alpha-nDCG  @20.  For  the  primary  measure  alpha-nDCG  @10,  our  approach  of 
eliminating  duplicate  domains  from  the  result  lists  resulted  in  an  improvement  for  14  of  the  50  query 
topics,  while  a  decrease  in  accuracy  was  observed  for  8  of  the  query  topics,  with  the  remaining  28 
unaffected. 

Overall  the  benefits  of  the  naive  approach  taken  is  questionable  given  the  mixed  results  over 
the  varying  measures.  We  hypothesis  that  in  combination  with  other  techniques  such  as  document 
clustering  such  an  approach  would  prove  more  beneficial. 

4  Conclusions 

RMIT  submitted  several  runs  to  the  various  Web  Track  tasks.  The  two  “out  of  the  box”  Zettair 
adhoc  run  submissions  achieved  median  like  accuracy  which  is  to  be  expected  for  a  simple  term 
matching  based  approach.  This  result  forms  a  reasonable  baseline  for  more  advanced  techniques  in 
future  submissions  using  the  newer  large  collection  introduced  this  year. 

The  unique  domain  name  approach  taken  for  the  diversity  task  produced  minimal  changes  to 
the  accuracy  of  the  run  when  compared  to  the  baseline  run. 
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